ESQL: StringIndexOutOfBoundsException when parsing metadata

[alex-spies]

We got the following stack trace from Serverless:

        "java.lang.StringIndexOutOfBoundsException: Range [55, 55 + -54) out of bounds for length 161    
  \tat java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55)    
  \tat java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52)    
  \tat java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213)    
  \tat java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210)    
  \tat java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98)    
  \tat java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromIndexSize(Preconditions.java:118)    
  \tat java.base/jdk.internal.util.Preconditions.checkFromIndexSize(Preconditions.java:397)    
  \tat java.base/java.lang.String.checkBoundsOffCount(String.java:4938)    
  \tat java.base/java.lang.String.<init>(String.java:543)    
  \tat org.antlr.v4.runtime.CodePointCharStream$CodePoint8BitCharStream.getText(CodePointCharStream.java:160)    
  \tat org.antlr.v4.runtime.CommonToken.getText(CommonToken.java:170)    
  \tat org.elasticsearch.xpack.esql.parser.ParserUtils.nameOrPosition(ParserUtils.java:147)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlParser$ParametrizedTokenSource.nextToken(EsqlParser.java:233)    
  \tat org.antlr.v4.runtime.BufferedTokenStream.fetch(BufferedTokenStream.java:169)    
  \tat org.antlr.v4.runtime.BufferedTokenStream.sync(BufferedTokenStream.java:152)    
  \tat org.antlr.v4.runtime.BufferedTokenStream.nextTokenOnChannel(BufferedTokenStream.java:325)    
  \tat org.antlr.v4.runtime.CommonTokenStream.adjustSeekIndex(CommonTokenStream.java:70)    
  \tat org.antlr.v4.runtime.BufferedTokenStream.consume(BufferedTokenStream.java:137)    
  \tat org.antlr.v4.runtime.Parser.consume(Parser.java:571)    
  \tat org.antlr.v4.runtime.Parser.match(Parser.java:205)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.metadata(EsqlBaseParser.java:1412)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.indexPatternAndMetadataFields(EsqlBaseParser.java:1173)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.fromCommand(EsqlBaseParser.java:1042)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.sourceCommand(EsqlBaseParser.java:434)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.query(EsqlBaseParser.java:339)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlBaseParser.singleStatement(EsqlBaseParser.java:241)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlParser.invokeParser(EsqlParser.java:148)    
  \tat org.elasticsearch.xpack.esql.parser.EsqlParser.createStatement(EsqlParser.java:114)    
  \tat org.elasticsearch.xpack.esql.session.EsqlSession.parse(EsqlSession.java:321)    
  \tat org.elasticsearch.xpack.esql.session.EsqlSession.execute(EsqlSession.java:164)    
  \tat org.elasticsearch.xpack.esql.execution.PlanExecutor.lambda$esql$2(PlanExecutor.java:96)
...

Somehow, calling token.getText() causes an out of bounds when parsing, I think, FROM ... METADATA ....

The out of bounds is triggered inside code that was most recently changed in #122459, although I can't say if the latter PR caused this.

I didn't manage to repro this after trying for the better part of an hour.

@fang-xing-esql , I noticed that before #122459, EsqlParser.nextToken() used to check for

token.getType() == EsqlBaseLexer.NAMED_OR_POSITIONAL_PARAM

before attempting to call token.getText, whereas now we call token.getText() before checking the token type (in ParserUtils.nameOrPosition()`). That might be why we see this now and didn't run into this before.

[elasticsearchmachine]

Pinging @elastic/es-analytical-engine (Team:Analytics)

[fang-xing-esql]

I couldn't reproduce this on main, the offset of the EsqlBaseParser.metadata on main is different from the one in the original stacktrace, the others from EsqlBaseParser are the same. I'll keep an eye to see if it reoccurs.

org.elasticsearch.xpack.esql.parser.ParserUtils::nameOrPosition::150
org.elasticsearch.xpack.esql.parser.EsqlParser$ParametrizedTokenSource::nextToken::233
org.antlr.v4.runtime.BufferedTokenStream::fetch::169
org.antlr.v4.runtime.BufferedTokenStream::sync::152
org.antlr.v4.runtime.BufferedTokenStream::nextTokenOnChannel::325
org.antlr.v4.runtime.CommonTokenStream::adjustSeekIndex::70
org.antlr.v4.runtime.BufferedTokenStream::consume::137
org.antlr.v4.runtime.Parser::consume::571 
org.antlr.v4.runtime.Parser::match::205
org.elasticsearch.xpack.esql.parser.EsqlBaseParser::metadata::1486   ==> different from 1412
org.elasticsearch.xpack.esql.parser.EsqlBaseParser::indexPatternAndMetadataFields::1173
org.elasticsearch.xpack.esql.parser.EsqlBaseParser::fromCommand::1042
org.elasticsearch.xpack.esql.parser.EsqlBaseParser::sourceCommand::434
org.elasticsearch.xpack.esql.parser.EsqlBaseParser::query::339

[fang-xing-esql]

There is another occurrence of this issue.