Add index_options to semantic_text field mappings #119967

kderusso · 2025-01-10T16:15:38Z

Adds index_options support for semantic_text fields using dense models.

Example:

PUT _inference/text_embedding/my-e5-model
{
  "service": "elasticsearch",
  "service_settings": {
    "num_allocations": 1,
    "num_threads": 1,
    "model_id": ".multilingual-e5-small"
  }
}

PUT my-semantic-index
{
  "mappings": {
    "properties": {
      "inference_field": {
        "type": "semantic_text",
        "inference_id": "my-e5-model",
        "index_options": {
          "dense_vector": {
            "type": "bbq_hnsw",
            "ef_construction": 100
           }
        }
      }
    }
  }
}

elasticsearchmachine · 2025-01-10T16:38:05Z

Hi @kderusso, I've created a changelog YAML for you.

elasticsearchmachine · 2025-01-10T16:39:12Z

Pinging @elastic/es-search-relevance (Team:Search Relevance)

elasticsearchmachine · 2025-01-10T16:39:12Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-01-10T16:39:12Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

Mikep86

Good start to this! I have a bunch of comments, but they're mostly interrelated, so it's not as much as it seems.

docs/changelog/119967.yaml

docs/reference/mapping/types/semantic-text.asciidoc

...ugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextField.java

...e/src/yamlRestTest/resources/rest-api-spec/test/inference/10_semantic_text_field_mapping.yml

...ugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextField.java

This reverts commit b08e2a1.

This reverts commit aedfafe.

jimczi

I left one comment regarding validation but this feels very close @kderusso !

...nference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java

github-actions · 2025-06-17T11:57:54Z

🔍 Preview links for changed docs:

docs/reference/elasticsearch/mapping-reference/semantic-text.md

🔔 The preview site may take up to 3 minutes to finish building. These links will become live once it completes.

Samiul-TheSoccerFan

Nice implementation and tests, left a few nitpick comments.

docs/changelog/119967.yaml

...nce/src/test/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapperTests.java

jimczi

elasticsearchmachine · 2025-06-17T19:02:55Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 119967

* Add index_options parameter to semantic_text field mapping * Cleanup & tests * Update docs * Update docs/changelog/119967.yaml * Addressed some PR feedbak * Update yaml tests * Refactoring * Cleanup * Fix some tests * Hack in inferring text_embedding task type from index options * [CI] Auto commit changes from spotless * Fix error inferring model settings * Update docs * Update tests * Update docs/reference/mapping/types/semantic-text.asciidoc Co-authored-by: Mike Pellegrini <[email protected]> * Address some minor PR feedback * Remove partial model_settings with inferred task type * Cleanup * Remove unnecessary changes * Fix errors from merge * [CI] Auto commit changes from spotless * Cleanup * Checkpoint, saving changes before merge * Update parsing * [CI] Auto commit changes from spotless * Stash changes * Fix compile errors * [CI] Auto commit changes from spotless * Cleanup error * fix test * fix test * Fix another test * A bit of cleanup * Fix tests * Spotless * Respect index options if set over defaults * Cleanup * [CI] Auto commit changes from spotless * Support updating to compatible versions, add some cleanup and validation * Remove test that can't be done here - needs to be unit test * Add validation * Cleanup * Fix some yaml tests * [CI] Auto commit changes from spotless * Happy path early index validation works now; edge cases surrounding default BBQ remain * Always emit index options, even when using defaults * Minor cleanup * Fix test compilation failures * Fix some tests * Continue to iterate on test failures * Remove index options from inference field metadata as it is only needed at field creation time * Fix some tests * Remove transport version, no longer needed * Fix yaml tests * Add tests * IndexOptions don't need to implement Writeable * [CI] Auto commit changes from spotless * Refactor - move SemanticTextIndexOptions * Remove writeable * Move index_options parsing to semantic text field mapper * Cleanup * Fix test compilation issue * Cleanup * Remove whitespace * Remove writeables from index options * Disable merging null options? * Add docs * [CI] Auto commit changes from spotless * Revert "Disable merging null options?" This reverts commit 2ef8b1d. * Remove default serialization * Include default index option type to defaults * [CI] Auto commit changes from spotless * Go back to allowing null updateS * Cleanup * Fix validation error * Revert "Include default index option type to defaults" This reverts commit b08e2a1. * Update tests * Revert "Update tests" This reverts commit aedfafe. * Better fix for null inputs * Remove redundant merge validation --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Mike Pellegrini <[email protected]> (cherry picked from commit 813814b) # Conflicts: # docs/reference/elasticsearch/mapping-reference/semantic-text.md # server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java # x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java # x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapperTests.java

kderusso · 2025-06-17T19:18:31Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

kderusso · 2025-06-18T12:04:25Z

Had to create a manual backport: #129626

…129626) * Add index_options to semantic_text field mappings (#119967) * Add index_options parameter to semantic_text field mapping * Cleanup & tests * Update docs * Update docs/changelog/119967.yaml * Addressed some PR feedbak * Update yaml tests * Refactoring * Cleanup * Fix some tests * Hack in inferring text_embedding task type from index options * [CI] Auto commit changes from spotless * Fix error inferring model settings * Update docs * Update tests * Update docs/reference/mapping/types/semantic-text.asciidoc Co-authored-by: Mike Pellegrini <[email protected]> * Address some minor PR feedback * Remove partial model_settings with inferred task type * Cleanup * Remove unnecessary changes * Fix errors from merge * [CI] Auto commit changes from spotless * Cleanup * Checkpoint, saving changes before merge * Update parsing * [CI] Auto commit changes from spotless * Stash changes * Fix compile errors * [CI] Auto commit changes from spotless * Cleanup error * fix test * fix test * Fix another test * A bit of cleanup * Fix tests * Spotless * Respect index options if set over defaults * Cleanup * [CI] Auto commit changes from spotless * Support updating to compatible versions, add some cleanup and validation * Remove test that can't be done here - needs to be unit test * Add validation * Cleanup * Fix some yaml tests * [CI] Auto commit changes from spotless * Happy path early index validation works now; edge cases surrounding default BBQ remain * Always emit index options, even when using defaults * Minor cleanup * Fix test compilation failures * Fix some tests * Continue to iterate on test failures * Remove index options from inference field metadata as it is only needed at field creation time * Fix some tests * Remove transport version, no longer needed * Fix yaml tests * Add tests * IndexOptions don't need to implement Writeable * [CI] Auto commit changes from spotless * Refactor - move SemanticTextIndexOptions * Remove writeable * Move index_options parsing to semantic text field mapper * Cleanup * Fix test compilation issue * Cleanup * Remove whitespace * Remove writeables from index options * Disable merging null options? * Add docs * [CI] Auto commit changes from spotless * Revert "Disable merging null options?" This reverts commit 2ef8b1d. * Remove default serialization * Include default index option type to defaults * [CI] Auto commit changes from spotless * Go back to allowing null updateS * Cleanup * Fix validation error * Revert "Include default index option type to defaults" This reverts commit b08e2a1. * Update tests * Revert "Update tests" This reverts commit aedfafe. * Better fix for null inputs * Remove redundant merge validation --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Mike Pellegrini <[email protected]> (cherry picked from commit 813814b) # Conflicts: # docs/reference/elasticsearch/mapping-reference/semantic-text.md # server/src/main/java/org/elasticsearch/index/mapper/vectors/DenseVectorFieldMapper.java # x-pack/plugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java # x-pack/plugin/inference/src/test/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapperTests.java * Fix errors in backport merge

Add index_options parameter to semantic_text field mapping

67b7623

elasticsearchmachine added the v9.0.0 label Jan 10, 2025

kderusso force-pushed the kderusso/semantic-text-index-options branch from e096a61 to 342d769 Compare January 10, 2025 16:17

Cleanup & tests

d822301

kderusso force-pushed the kderusso/semantic-text-index-options branch from 342d769 to d822301 Compare January 10, 2025 16:29

Update docs

251d22c

kderusso added >enhancement auto-backport Automatically create backport pull requests when merged :SearchOrg/Relevance Label for the Search (solution/org) Relevance team v8.18.0 labels Jan 10, 2025

Update docs/changelog/119967.yaml

8724cce

kderusso added the :Search Relevance/Search Catch all for Search Relevance label Jan 10, 2025

Merge branch 'main' into kderusso/semantic-text-index-options

6445a44

kderusso marked this pull request as ready for review January 10, 2025 16:38

kderusso requested review from jimczi, Mikep86 and a team January 10, 2025 16:39

Merge branch 'main' into kderusso/semantic-text-index-options

26de9d3

Mikep86 reviewed Jan 10, 2025

View reviewed changes

Addressed some PR feedbak

948d596

jimczi reviewed Jan 13, 2025

View reviewed changes

...ugin/inference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextField.java Outdated Show resolved Hide resolved

kderusso added 5 commits January 14, 2025 14:25

Update yaml tests

860ebc4

Refactoring

07284f5

Cleanup

3c1f6e1

Merge main into kderusso/semantic-text-index-options

b4f45ea

Fix some tests

5d2f48b

kderusso added 3 commits June 16, 2025 15:54

Fix validation error

e993671

Revert "Include default index option type to defaults"

b843047

This reverts commit b08e2a1.

Update tests

aedfafe

kderusso requested review from jimczi and Mikep86 June 16, 2025 20:16

kderusso added 2 commits June 16, 2025 16:52

Revert "Update tests"

062eeac

This reverts commit aedfafe.

Better fix for null inputs

65a3d02

jimczi reviewed Jun 16, 2025

View reviewed changes

...nference/src/main/java/org/elasticsearch/xpack/inference/mapper/SemanticTextFieldMapper.java Outdated Show resolved Hide resolved

Merge branch 'main' into kderusso/semantic-text-index-options

fa22c56

Remove redundant merge validation

57bd949

kderusso requested review from jimczi, benwtrent and leemthompo June 17, 2025 12:59

Merge branch 'main' into kderusso/semantic-text-index-options

2db090c

Samiul-TheSoccerFan approved these changes Jun 17, 2025

View reviewed changes

jimczi approved these changes Jun 17, 2025

View reviewed changes

Merge branch 'main' into kderusso/semantic-text-index-options

a6c04a5

kderusso merged commit 813814b into elastic:main Jun 17, 2025
27 checks passed

elasticsearchmachine added the backport pending label Jun 17, 2025

This was referenced Jun 17, 2025

[8.19] Add index_options to semantic_text field mappings (#119967) #129587

Closed

[8.19] Add index_options to semantic_text field mappings (#119967) #129626

Merged

Add index_options to semantic_text field mappings #119967

Add index_options to semantic_text field mappings #119967

Uh oh!

Conversation

kderusso commented Jan 10, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

elasticsearchmachine commented Jan 10, 2025

Uh oh!

elasticsearchmachine commented Jan 10, 2025

Uh oh!

elasticsearchmachine commented Jan 10, 2025

Uh oh!

elasticsearchmachine commented Jan 10, 2025

Uh oh!

Mikep86 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

github-actions bot commented Jun 17, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Samiul-TheSoccerFan left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jimczi left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

elasticsearchmachine commented Jun 17, 2025

💔 Backport failed

Uh oh!

kderusso commented Jun 17, 2025

💚 All backports created successfully

Questions ?

Uh oh!

kderusso commented Jun 18, 2025

Uh oh!

Uh oh!

kderusso commented Jan 10, 2025 •

edited

Loading

github-actions bot commented Jun 17, 2025 •

edited

Loading