Throw a 400 when sorting for all types of range fields

[drempapis]

In main, sorting on Range Fields Triggers AssertionError and Unexpected Behavior in ES.

When assertions are enabled, attempting to sort on certain range field types, such as integer_range, long_range, float_range, double_range, or ip_range, results in a 400. This occurs due to a search_phase_execution_exception caused by an illegal_argument_exception, which is ultimately rooted in an AssertionError during Lucene's byte-to-UTF8 formatting.

This issue occurs when the code attempts to format a BytesRef value returned from a range field sort as a human-readable string. Still, the operation fails because the data is not intended to be interpreted as UTF-8. The relevant assertion expects sort values to be of known types (String, Number, Boolean, or Map) and throws when raw bytes are encountered instead.

When assertions are disabled, the sort operation completes without failure; however, it returns non-human-readable byte sequences (e.g., "\u0001��") as sort values. This behavior is considered unreliable and unsuitable for use in production environments.

For the date_range field type, the behavior is slightly different. When assertions are enabled, execution fails with the following error:java.lang.AssertionError: [1 a9 94 1f 29 7c 0 a9 9b 76 da a7 ff] was not formatted.

When assertions are disabled, and the size parameter restricts the number of returned documents to fewer than the total available, a 500 Internal Server Error is triggered, caused by an unsupported_operation_exception.

This PR disables sorting for all range field types, aligning their behavior with other non-sortable field types (e.g., geo types). When a sort operation is attempted on a range field, Elasticsearch now returns a 400 response with a clear message indicating that sorting is not supported for the given field, which is of type range field.

solves #122358

[elasticsearchmachine]

Hi @drempapis, I've created a changelog YAML for you.

[elasticsearchmachine]

Pinging @elastic/es-search-foundations (Team:Search Foundations)

[javanna]

We probably need to update the docs as well to highlight this discrepancy, I believe the sorting page does not specify that certain field types don't support sorting.

[javanna]

how do we do the same for other non sortable field types? Are there ways to avoid an instance of check perhaps?

[drempapis]

Thank you, @javanna. You're right, using instanceof can lead to fragile and hard-to-maintain code in certain cases. In this instance, my approach was primarily influenced by the method of application, as instanceof is used in several places throughout the code.

Following the behavior observed with geo_point, where calling AbstractPointIndexFieldData#sortField results in an IllegalArgumentException, I updated the RangeFieldMapper#fielddataBuilder implementation. Specifically, we now return an IndexFieldData.Builder that overrides the sortField method to customize its behavior for all range types.

[drempapis]

@javanna, I was considering updating the documentation in a separate PR. Would it be preferable to include it as part of this one instead?"

[javanna]

LGTM, assuming @iverase is onboard with it too

Thanks for fixing this!

[javanna]

I like this solution, much better than the instance of. Those should be removed rather than being a precedent for adding more over time :)

[drempapis]

I fully agree, thank you for the review!

[elasticsearchmachine]

💚 Backport successful

Status	Branch	Result
✅	9.0
✅	9.1