Skip to content

ArrayIndexOutOfBoundsException in FixedBitSet.set during query #126939

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Closed
jordan-powers opened this issue Apr 16, 2025 · 8 comments
Closed

ArrayIndexOutOfBoundsException in FixedBitSet.set during query #126939

jordan-powers opened this issue Apr 16, 2025 · 8 comments
Assignees
@ChrisHegarty
Labels
blocker >bug :Search Relevance/Search Catch all for Search Relevance Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch

Comments

@jordan-powers
Copy link
Contributor

Looks like it happens during scoring.

Full stack trace:

org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
	at [email protected]/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:668)
	at [email protected]/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:873)
	at [email protected]/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:706)
	at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
	at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
	at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.lang.ArrayIndexOutOfBoundsException: Index -33 out of bounds for length 64
	at [email protected]/org.apache.lucene.util.FixedBitSet.set(FixedBitSet.java:283)
	at [email protected]/org.apache.lucene.search.DocIdSetIterator.intoBitSet(DocIdSetIterator.java:268)
	at [email protected]/org.apache.lucene.search.comparators.TermOrdValComparator$CompetitiveIterator.intoBitSet(TermOrdValComparator.java:539)
	at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindowUsingBitSet(DenseConjunctionBulkScorer.java:242)
	at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:210)
	at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
	at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
	at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
	at [email protected]/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:369)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:336)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
	... 15 more
@jordan-powers jordan-powers added :SearchOrg/Relevance Label for the Search (solution/org) Relevance team >bug labels Apr 16, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/search-eng (Team:SearchOrg)

@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/search-relevance (Team:Search - Relevance)

@kderusso kderusso added the :Search Relevance/Search Catch all for Search Relevance label Apr 16, 2025
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/es-search-relevance (Team:Search Relevance)

@jordan-powers
Copy link
Contributor Author

Got another stack trace that looks related. Let me know if this is actually unrelated and I'll open a new, separate issue.

org.elasticsearch.search.query.QueryPhaseExecutionException: Query Failed [Failed to execute main query]
	at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:237)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.executeQuery(QueryPhase.java:143)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.execute(QueryPhase.java:70)
	at [email protected]/org.elasticsearch.indices.IndicesService.lambda$loadIntoContext$38(IndicesService.java:1627)
	at [email protected]/org.elasticsearch.indices.IndicesService.lambda$cacheShardLevelResult$39(IndicesService.java:1693)
	at [email protected]/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:165)
	at [email protected]/org.elasticsearch.indices.IndicesRequestCache$Loader.load(IndicesRequestCache.java:148)
	at [email protected]/org.elasticsearch.common.cache.Cache.computeIfAbsent(Cache.java:440)
	at [email protected]/org.elasticsearch.indices.IndicesRequestCache.getOrCompute(IndicesRequestCache.java:111)
	at [email protected]/org.elasticsearch.indices.IndicesService.cacheShardLevelResult(IndicesService.java:1699)
	at [email protected]/org.elasticsearch.indices.IndicesService.loadIntoContext(IndicesService.java:1621)
	at [email protected]/org.elasticsearch.search.SearchService.loadOrExecuteQueryPhase(SearchService.java:666)
	at [email protected]/org.elasticsearch.search.SearchService.executeQueryPhase(SearchService.java:873)
	at [email protected]/org.elasticsearch.search.SearchService.lambda$executeQueryPhase$7(SearchService.java:706)
	at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:79)
	at [email protected]/org.elasticsearch.action.ActionRunnable$3.accept(ActionRunnable.java:76)
	at [email protected]/org.elasticsearch.action.ActionRunnable$4.doRun(ActionRunnable.java:101)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at [email protected]/org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:34)
	at [email protected]/org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:1044)
	at [email protected]/org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:27)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1095)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:619)
	at java.base/java.lang.Thread.run(Thread.java:1447)
Caused by: java.lang.IndexOutOfBoundsException: Range [0, -14) out of bounds for length 4096
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
	at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
	at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
	at [email protected]/org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
	at [email protected]/org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
	at [email protected]/org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
	at [email protected]/org.apache.lucene.search.TotalHitCountCollector$1.collect(TotalHitCountCollector.java:69)
	at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:208)
	at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
	at [email protected]/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
	at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
	at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
	at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
	at [email protected]/org.apache.lucene.search.TaskExecutor.invokeAll(TaskExecutor.java:111)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:369)
	at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:336)
	at [email protected]/org.elasticsearch.search.query.QueryPhase.addCollectorsAndSearch(QueryPhase.java:212)
	... 23 more
	Suppressed: java.lang.IndexOutOfBoundsException: Range [0, -336) out of bounds for length 4096
		at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
		at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
		at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
		at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
		at [email protected]/org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
		at [email protected]/org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
		at [email protected]/org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
		at [email protected]/org.apache.lucene.search.TotalHitCountCollector$1.collect(TotalHitCountCollector.java:69)
		at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:208)
		at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
		at [email protected]/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
		at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
		at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
		at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
		at [email protected]/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
		... 6 more
	Suppressed: java.lang.IndexOutOfBoundsException: Range [0, -2) out of bounds for length 4096
		at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:100)
		at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromToIndex(Preconditions.java:112)
		at java.base/jdk.internal.util.Preconditions.checkFromToIndex(Preconditions.java:349)
		at java.base/java.util.Objects.checkFromToIndex(Objects.java:391)
		at [email protected]/org.apache.lucene.util.FixedBitSet.cardinality(FixedBitSet.java:212)
		at [email protected]/org.apache.lucene.search.DISIDocIdStream.count(DISIDocIdStream.java:63)
		at [email protected]/org.apache.lucene.search.DocIdStream.count(DocIdStream.java:50)
		at [email protected]/org.apache.lucene.search.TotalHitCountCollector$1.collect(TotalHitCountCollector.java:69)
		at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.scoreWindow(DenseConjunctionBulkScorer.java:208)
		at [email protected]/org.apache.lucene.search.DenseConjunctionBulkScorer.score(DenseConjunctionBulkScorer.java:132)
		at [email protected]/org.apache.lucene.search.ReqExclBulkScorer.score(ReqExclBulkScorer.java:66)
		at [email protected]/org.elasticsearch.search.internal.CancellableBulkScorer.score(CancellableBulkScorer.java:46)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.searchLeaf(ContextIndexSearcher.java:460)
		at [email protected]/org.apache.lucene.search.IndexSearcher.search(IndexSearcher.java:809)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.search(ContextIndexSearcher.java:387)
		at [email protected]/org.elasticsearch.search.internal.ContextIndexSearcher.lambda$search$3(ContextIndexSearcher.java:365)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:328)
		at [email protected]/org.apache.lucene.search.TaskExecutor$Task.run(TaskExecutor.java:173)
		at [email protected]/org.apache.lucene.search.TaskExecutor.lambda$invokeAll$1(TaskExecutor.java:98)
		... 6 more

@javanna javanna removed the :SearchOrg/Relevance Label for the Search (solution/org) Relevance team label Apr 17, 2025
@elasticsearchmachine elasticsearchmachine added the Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch label Apr 17, 2025
@benwtrent
Copy link
Member

related: apache/lucene#14517

I am not 100% sure these are directly a Lucene bug or not. However, it does seem that any assumption where a reference to iterators is kept needs to be revisited.

@jdcryans jdcryans mentioned this issue Apr 17, 2025
Open
@ChrisHegarty
Copy link
Contributor

ChrisHegarty commented Apr 18, 2025 •
edited
Loading

Possibly Lucene bug lucene/14522

The second of the stack traces show DISIDocIdStream::count, which should be fixed by the above Lucene PR.

@ChrisHegarty ChrisHegarty mentioned this issue Apr 18, 2025
Merged
@ChrisHegarty
Copy link
Contributor

ChrisHegarty commented Apr 20, 2025 •
edited
Loading

There are two similar, but separate, bugs identified by the stacktraces reported in this issue. Both have been fixed in Lucene:

  1. TermOrdValComparator$CompetitiveIterator.intoBitSet - fixed by Correct TermOrdValComparator competitive iterator intoBitSet implementation apache/lucene#14523

  2. DocIdStream.count - Fix DISIDocIdStream::count so that it does not try to count beyond max apache/lucene#14522

A Lucene 10.2.1 bugfix release has been suggested in order to get these fixes into a shipping release.

@ChrisHegarty ChrisHegarty self-assigned this Apr 22, 2025
@ChrisHegarty
Copy link
Contributor

ChrisHegarty commented Apr 22, 2025 •
edited
Loading

Fixed in Elasticsearch with a snapshot release of Lucene 10.2.1-SNAPSHOT, see #127125

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@ChrisHegarty ChrisHegarty

Labels
blocker >bug :Search Relevance/Search Catch all for Search Relevance Team:Search Relevance Meta label for the Search Relevance team in Elasticsearch
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@javanna @benwtrent @jordan-powers @elasticsearchmachine @ChrisHegarty @kderusso