Add max num_candidates as a dynamic index settings

[weizijun]

Now the num_candidates must be lower then 10000, In some case, use want to get more vectors.
I think the num_candidates can be a parameter like max_window_size, The use can change the default vaule.
I add a new setting named index.max_knn_num_candidates. It can dynamic modify.
The old PR (#125001) set as a index_option parameter in mappings.
The new PR change it into the index setting parameter.

[benwtrent]

much nicer! Could you also add some documentation in docs/reference/elasticsearch/index-settings/index-modules.md ?

[benwtrent]

I think eagerly validating like this is OK. However, KnnVectorQueryBuilder#doToQuery should also validate as its possible to provide a knn query that isn't executed through DFS.

[weizijun]

Yeah, I think the check in KnnVectorQueryBuilder#doToQuery is better, I will change the check code.

[weizijun]

done

[elasticsearchmachine]

Pinging @elastic/es-search-relevance (Team:Search Relevance)

[tteofili]

I wonder about the use case here. it sounds like the ask here is to consider a vast number of candidates, I wonder if this wouldn't be best served by an explicit brute force search?

[weizijun]

I wonder about the use case here. it sounds like the ask here is to consider a vast number of candidates, I wonder if this wouldn't be best served by an explicit brute force search?

If the amount of shard data is too large, brute force search will be slow. You can set a larger number of candidates to obtain more vector data.

[weizijun]

much nicer! Could you also add some documentation in docs/reference/elasticsearch/index-settings/index-modules.md ?

done

[benwtrent]

@weizijun is the use-case here aggregating a large number of vector matches? Or an attempt to have very very accurate results from large graphs?

HNSW performance gets worse and worse the more vectors you search and wish to return.

I am just wanting to better understand the reason for this change.

[weizijun]

HNSW performance gets worse and worse the more vectors you search and wish to return.

I am just wanting to better understand the reason for this change.

10000 is a hard limit, when the vector data is large, in some data mining cases, users want to get more docs, 10000 docs is not enough. Self-driving customers want to find more images that is similar to the query image, it will raise the limit of num_candidates.

And I think there is a more solution, the FloatVectorSimilarityQuery in lucene may be a solution, it will collect all docs which similarity is that the min similarity. The min similarity parameter in lucene FloatVectorSimilarityQuery is not the same with which define in elasticsearch knn query.

[benwtrent]

Self-driving customers want to find more images that is similar to the query image, it will raise the limit of num_candidates.

Increasing num_candidates doesn't fix this. You still won't be able to get more than 10k total nearest neighbors in the final result set due to the limitation of k.

So the actual use case is wanting to search over more than 10k total hits?

This also proves difficult as the maximum docs allowed (e.g. the total allowed to search over with from+size) is 10k.

I don't see how you can actually search over more than 10k nearest neighbors without significant changes to other parts of Elasticsearch.

The min similarity parameter in lucene FloatVectorSimilarityQuery is not the same with which define in elasticsearch knn query.

Correct, this is because the similarity query in Lucene as it is currently designed is syntactic sugar for "Please do a complete brute force query and filter on similarity". Very often it reverts to searching the entire index as HNSW is not designed for this type of query.

[weizijun]

Increasing num_candidates doesn't fix this. You still won't be able to get more than 10k total nearest neighbors in the final result set due to the limitation of k.

So the actual use case is wanting to search over more than 10k total hits?

This also proves difficult as the maximum docs allowed (e.g. the total allowed to search over with from+size) is 10k.

I don't see how you can actually search over more than 10k nearest neighbors without significant changes to other parts of Elasticsearch.

When the num_candidates raise, it will also rise the max_window_size, or use the scroll to get more docs.

[benwtrent]

When the num_candidates raise, it will also rise the max_window_size, or use the scroll to get more docs.

max_window_size if you are referring to rescore parameters or the retrievers frame work, again, those are all limited by the actual results you can get, which is 10k.

The only way I could see somebody getting MORE than 10k total docs is via scroll.

The only usages I see (without significant work elsewhere) for increasing num_candidates above 10,000 are:

• Aggregation results
• Hyper accurate results on very large graphs.

However, neither of these seem to be the reason for this change.

I am possibly confused here :(.