Pinned retriever #126401

mridula-s109 · 2025-04-07T12:24:42Z

Pinned Retriever for Elasticsearch

This PR introduces a new pinned retriever for Elasticsearch, available under the search-business-rules plugin.

The pinned retriever allows users to explicitly pin specific documents (via ids or docs) so they always appear at the top of the search results, regardless of their organic relevance score. It is implemented as a compound retriever that rewrites internally to a PinnedQueryBuilder.

Key Features

Pin Documents by ID or Full Doc: Pin specific documents either by their _id or by providing inline doc content.
Custom Explanation Support: When used with "explain": true, the search response now includes a custom _explanation field for pinned results.
Example:
```
"description": "Pinned document by ids, original explanation:"
```

Example API Request

POST my-index/_search
{
  "explain": true,
  "retriever": {
    "pinned": {
      "ids": ["1", "2"],
      "retriever": {
        "standard": {
          "query": {
            "match": {
              "title": "machine learning"
            }
          }
        }
      }
    }
  }
}

Sample Response

"_explanation": {
  "value": 1.7014124e+38,
  "description": "Pinned document, original explanation:",
  "details": [
    {
      "value": 1,
      "description": "doc [0] with an original score of [1.7014124E38] is at rank [1] from the following source queries.",
      "details": [
        {
          // additional explanation details...
        }
      ]
    }
  ]
}

elasticsearchmachine · 2025-04-07T20:58:01Z

Hi @mridula-s109, I've created a changelog YAML for you.

elasticsearchmachine · 2025-04-07T21:43:20Z

Pinging @elastic/search-eng (Team:SearchOrg)

elasticsearchmachine · 2025-04-07T21:43:21Z

Pinging @elastic/search-relevance (Team:Search - Relevance)

kderusso

Quick pass through, I'm surprised this works without registering the retriever?

server/src/main/java/org/elasticsearch/TransportVersions.java

...es/src/yamlRestTest/resources/rest-api-spec/test/searchbusinessrules/10_pinned_retriever.yml

elasticsearchmachine · 2025-04-08T10:43:02Z

Hi @mridula-s109, I've created a changelog YAML for you.

* Fix compilation error * Remove SearchPlugin from META-INF * Remove duplicate FeatureSpecification * Add service as test resource * Move to test * More file moving * Remove from module-info, remove file * Make search business rules plugin extensible * fix ent search plugin * Delete dup files * [CI] Auto commit changes from spotless * Add module info back in --------- Co-authored-by: elasticsearchmachine <[email protected]>

kderusso

Some hopefully final feedback! Thanks for your iterations on this.
As a best practice it would be good to request review from the Search Relevance team so everyone on the team is aware of this PR.

Awaiting CI and final changes but this is really close!

...plugin/rank-rrf/src/yamlRestTest/resources/rest-api-spec/test/rrf/950_pinned_interaction.yml

...ess-rules/src/main/java/org/elasticsearch/xpack/searchbusinessrules/SearchBusinessRules.java

.../main/java/org/elasticsearch/xpack/searchbusinessrules/retriever/PinnedRetrieverBuilder.java

.../src/yamlRestTest/resources/rest-api-spec/test/search-business-rules/10_pinned_retriever.yml

…, as i had made changes related to the cluster error

mridula-s109 · 2025-04-30T19:59:08Z

Some hopefully final feedback! Thanks for your iterations on this. As a best practice it would be good to request review from the Search Relevance team so everyone on the team is aware of this PR.

Awaiting CI and final changes but this is really close!

I have resolved all the comments @kderusso , hopefully the ci build passes through and everything works as intended.

ioanatia

Looks good. I tested it locally.
Checked the comments from @kderusso from the last review and it looks to me that all have been addressed.

ioanatia · 2025-05-01T11:13:22Z

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -231,6 +231,7 @@ static TransportVersion def(int id) {
    public static final TransportVersion INTRODUCE_FAILURES_LIFECYCLE = def(9_065_0_00);
    public static final TransportVersion PROJECT_METADATA_SETTINGS = def(9_066_00_0);
    public static final TransportVersion AGGREGATE_METRIC_DOUBLE_BLOCK = def(9_067_00_0);
+    public static final TransportVersion PINNED_RETRIEVER = def(9_068_0_00);


just a side note - but when we backport this to 8.19, we need to pay extra attention on how we backport this transport version

Yeah @mridula-s109 - we will create a PINNED_RETRIEVER_8_19 transport version

kderusso

Looks great! Thanks for all the iterations!

kderusso · 2025-05-01T12:48:22Z

server/src/main/java/org/elasticsearch/TransportVersions.java

@@ -231,6 +231,7 @@ static TransportVersion def(int id) {
    public static final TransportVersion INTRODUCE_FAILURES_LIFECYCLE = def(9_065_0_00);
    public static final TransportVersion PROJECT_METADATA_SETTINGS = def(9_066_00_0);
    public static final TransportVersion AGGREGATE_METRIC_DOUBLE_BLOCK = def(9_067_00_0);
+    public static final TransportVersion PINNED_RETRIEVER = def(9_068_0_00);


Yeah @mridula-s109 - we will create a PINNED_RETRIEVER_8_19 transport version

elasticsearchmachine · 2025-05-01T12:51:41Z

💔 Backport failed

Status	Branch	Result
❌	8.19	Commit could not be cherrypicked due to conflicts

You can use sqren/backport to manually backport by running backport --upstream elastic/elasticsearch --pr 126401

mridula-s109 · 2025-05-02T13:39:16Z

💚 All backports created successfully

Status	Branch	Result
✅	8.19

Questions ?

Please refer to the Backport tool documentation

* Created PinnedQueryRetriever Builder * Added retriever build test file * Changed the test to accept all the retrievers as its not licensed * [CI] Auto commit changes from spotless * Added integration test and fixed the code style issues * Fixed merge issues * Added cluster test feature to the integration tests * Update docs/changelog/126401.yaml * Registered retriever plugin * Enhanced changelog description * Added validation to the constructor creation * Introduced validations in the code and incorporated in tests - compiling version * Included PinnedRankDoc to enhance the explain query, validations for inputs anfd sorting. Fixed the tests after these changes * Working on improving integration test and introducing cluster features * Removed cluster test temporarily * Got pinnedretriever yaml to working state without cluster * [CI] Auto commit changes from spotless * Resolved no source yaml error * Cluster loaded successfully * [CI] Auto commit changes from spotless * Fixed integration test * Made validate sort less strict * Included shareddoc sorting in the validate sort * All yaml issues resolved * Explanation yaml files added * Trying to add clustering to the test * [CI] Auto commit changes from spotless * Removed explicit feature specification * Deleted the empty file * Put the node feature in the proper place * Added additional validation in ids and docs * Cleaned the create pinned query validation * made unit tests robust * Remove query.es from version control * Remove result.json from version control * [CI] Auto commit changes from spotless * Everything except explanation is fixed * added duplicate doc test * Applied checkstyle fix, spotless and a failing yaml * Improvements based on PR comments * [CI] Auto commit changes from spotless * Modified the unit test to accomodate the change in createPinnedQuery * Split the yaml to test for basic and trial, cleanedup and acted on all the comments * Remove result.json and query.es from version control * Removed unnecessary comments * Removed redundant file * Fixing CI build error * [CI] Auto commit changes from spotless * Removed pinnedBy as it wasnt necessary * Removed unnecessary ToXContent Information * Fixed transport version charges and cleaned up null checks: " * [CI] Auto commit changes from spotless * Retriever status changed to 9.1 version * cleaned up 2 yamltestsuite for different licenses * Trying to get the clustering works * Cleaned up the yaml clustering and also reorganised the yaml tests * Unnecessary file introduction removed * Reverted the plugins to the previous state as the changes werent necessary * Removed unnecessary transport versioning from pinnedrankdoc * reverted * [CI] Auto commit changes from spotless * Edited the SearchBusinessRules to remove the class from getFeatures * [CI] Auto commit changes from spotless * Cleaned up Pinned retriever to allow only id or docs * Added more test to the pinned retriever * did spotless * Introduced new transport versioning * Cleaning it up * Resolved validate module error * BWT issues * [CI] Auto commit changes from spotless * fix NPE occuring in ci * trying to fix duplicate feature issue * [CI] Auto commit changes from spotless * modified the tests * Playing with pinned retriever CI (elastic#127530) * Fix compilation error * Remove SearchPlugin from META-INF * Remove duplicate FeatureSpecification * Add service as test resource * Move to test * More file moving * Remove from module-info, remove file * Make search business rules plugin extensible * fix ent search plugin * Delete dup files * [CI] Auto commit changes from spotless * Add module info back in --------- Co-authored-by: elasticsearchmachine <[email protected]> * top document no pinned * Removing this to see if it works without this transport version check, as i had made changes related to the cluster error * Fixed the retriever builder comments and error message * removed the unnencessary explain * Fixed all the tests in the yaml file * added extra test case * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Kathleen DeRusso <[email protected]> (cherry picked from commit b742b02) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java

* Pinned retriever (#126401) * Created PinnedQueryRetriever Builder * Added retriever build test file * Changed the test to accept all the retrievers as its not licensed * [CI] Auto commit changes from spotless * Added integration test and fixed the code style issues * Fixed merge issues * Added cluster test feature to the integration tests * Update docs/changelog/126401.yaml * Registered retriever plugin * Enhanced changelog description * Added validation to the constructor creation * Introduced validations in the code and incorporated in tests - compiling version * Included PinnedRankDoc to enhance the explain query, validations for inputs anfd sorting. Fixed the tests after these changes * Working on improving integration test and introducing cluster features * Removed cluster test temporarily * Got pinnedretriever yaml to working state without cluster * [CI] Auto commit changes from spotless * Resolved no source yaml error * Cluster loaded successfully * [CI] Auto commit changes from spotless * Fixed integration test * Made validate sort less strict * Included shareddoc sorting in the validate sort * All yaml issues resolved * Explanation yaml files added * Trying to add clustering to the test * [CI] Auto commit changes from spotless * Removed explicit feature specification * Deleted the empty file * Put the node feature in the proper place * Added additional validation in ids and docs * Cleaned the create pinned query validation * made unit tests robust * Remove query.es from version control * Remove result.json from version control * [CI] Auto commit changes from spotless * Everything except explanation is fixed * added duplicate doc test * Applied checkstyle fix, spotless and a failing yaml * Improvements based on PR comments * [CI] Auto commit changes from spotless * Modified the unit test to accomodate the change in createPinnedQuery * Split the yaml to test for basic and trial, cleanedup and acted on all the comments * Remove result.json and query.es from version control * Removed unnecessary comments * Removed redundant file * Fixing CI build error * [CI] Auto commit changes from spotless * Removed pinnedBy as it wasnt necessary * Removed unnecessary ToXContent Information * Fixed transport version charges and cleaned up null checks: " * [CI] Auto commit changes from spotless * Retriever status changed to 9.1 version * cleaned up 2 yamltestsuite for different licenses * Trying to get the clustering works * Cleaned up the yaml clustering and also reorganised the yaml tests * Unnecessary file introduction removed * Reverted the plugins to the previous state as the changes werent necessary * Removed unnecessary transport versioning from pinnedrankdoc * reverted * [CI] Auto commit changes from spotless * Edited the SearchBusinessRules to remove the class from getFeatures * [CI] Auto commit changes from spotless * Cleaned up Pinned retriever to allow only id or docs * Added more test to the pinned retriever * did spotless * Introduced new transport versioning * Cleaning it up * Resolved validate module error * BWT issues * [CI] Auto commit changes from spotless * fix NPE occuring in ci * trying to fix duplicate feature issue * [CI] Auto commit changes from spotless * modified the tests * Playing with pinned retriever CI (#127530) * Fix compilation error * Remove SearchPlugin from META-INF * Remove duplicate FeatureSpecification * Add service as test resource * Move to test * More file moving * Remove from module-info, remove file * Make search business rules plugin extensible * fix ent search plugin * Delete dup files * [CI] Auto commit changes from spotless * Add module info back in --------- Co-authored-by: elasticsearchmachine <[email protected]> * top document no pinned * Removing this to see if it works without this transport version check, as i had made changes related to the cluster error * Fixed the retriever builder comments and error message * removed the unnencessary explain * Fixed all the tests in the yaml file * added extra test case * [CI] Auto commit changes from spotless --------- Co-authored-by: elasticsearchmachine <[email protected]> Co-authored-by: Kathleen DeRusso <[email protected]> (cherry picked from commit b742b02) # Conflicts: # server/src/main/java/org/elasticsearch/TransportVersions.java * Fixed the compile issue * changed pinned retriever name

In elastic#127623 we backported elastic#127299 and added a backport transport version for it - `ESQL_AGGREGATE_METRIC_DOUBLE_BLOCK_8_19` aka `8_841_0_24`. This brings that version forwards to `main` and adds support for parsing streams with that version. In elastic#127639 we backported elastic#126401 and added a backport transport version for it - `PINNED_RETRIEVER_8_19` aka `8_841_0_23`. This brings that version forwards to `main` and adds support for parsing streams with that versions. In elastic#127633 we a claimed a backport transport version to backport elastic#127314 - `INTRODUCE_FAILURES_LIFECYCLE_BACKPORT_8_19` aka `8_841_0_23`. That's the same versions as `PINNED_RETRIEVER_8_19`. It's just that this one is in `main` and `PINNED_RETRIEVER_8_19` is in `8.19`. To allow me to bring `PINNED_RETRIEVER_8_19` for wards I've had to revert elastic#127633. Closes elastic#127667

In #127623 we backported #127299 and added a backport transport version for it - `ESQL_AGGREGATE_METRIC_DOUBLE_BLOCK_8_19` aka `8_841_0_24`. This brings that version forwards to `main` and adds support for parsing streams with that version. In #127639 we backported #126401 and added a backport transport version for it - `PINNED_RETRIEVER_8_19` aka `8_841_0_23`. This brings that version forwards to `main` and adds support for parsing streams with that versions. In #127633 we a claimed a backport transport version to backport #127314 - `INTRODUCE_FAILURES_LIFECYCLE_BACKPORT_8_19` aka `8_841_0_23`. That's the same versions as `PINNED_RETRIEVER_8_19`. It's just that this one is in `main` and `PINNED_RETRIEVER_8_19` is in `8.19`. To allow me to bring `PINNED_RETRIEVER_8_19` for wards I've had to revert #127633. Closes #127667

mridula-s109 added 3 commits April 7, 2025 09:23

Created PinnedQueryRetriever Builder

84b6ed7

Added retriever build test file

4c6f284

Changed the test to accept all the retrievers as its not licensed

18bf6e4

elasticsearchmachine added the v9.1.0 label Apr 7, 2025

elasticsearchmachine and others added 4 commits April 7, 2025 12:32

[CI] Auto commit changes from spotless

8bab3d2

Added integration test and fixed the code style issues

2cfcdc1

Fixed merge issues

760415d

Merge branch 'main' into pinned-retriever

9bc751a

mridula-s109 added >enhancement auto-backport Automatically create backport pull requests when merged :SearchOrg/Relevance Label for the Search (solution/org) Relevance team labels Apr 7, 2025

github-actions bot deployed to docs-preview April 7, 2025 20:58 View deployment

github-actions bot deployed to docs-preview April 7, 2025 21:17 View deployment

github-actions bot deployed to docs-preview April 7, 2025 21:42 View deployment

mridula-s109 marked this pull request as ready for review April 7, 2025 21:42

mridula-s109 enabled auto-merge (squash) April 7, 2025 21:43

mridula-s109 requested review from kderusso and a team April 7, 2025 21:44

kderusso reviewed Apr 8, 2025

View reviewed changes

server/src/main/java/org/elasticsearch/TransportVersions.java Outdated Show resolved Hide resolved

...es/src/yamlRestTest/resources/rest-api-spec/test/searchbusinessrules/10_pinned_retriever.yml Outdated Show resolved Hide resolved

mridula-s109 marked this pull request as draft April 8, 2025 09:44

auto-merge was automatically disabled April 8, 2025 09:44
Pull request was converted to draft

mridula-s109 force-pushed the pinned-retriever branch from f162bac to 9bc751a Compare April 8, 2025 09:53

mridula-s109 added 3 commits April 8, 2025 10:55

Merge branch 'main' into pinned-retriever

63ac692

Added cluster test feature to the integration tests

eec5f60

Update docs/changelog/126401.yaml

69e14cc

github-actions bot deployed to docs-preview April 8, 2025 10:43 View deployment

kderusso and others added 2 commits April 30, 2025 14:17

Merge branch 'main' into pinned-retriever

e86c6e0

kderusso reviewed Apr 30, 2025

View reviewed changes

mridula-s109 requested a review from a team April 30, 2025 17:27

mridula-s109 force-pushed the pinned-retriever branch from d1e4be8 to e86c6e0 Compare April 30, 2025 18:29

mridula-s109 added 7 commits April 30, 2025 19:39

top document no pinned

cbb8ab1

Removing this to see if it works without this transport version check…

9bfcb4b

…, as i had made changes related to the cluster error

Fixed the retriever builder comments and error message

227011b

removed the unnencessary explain

84608c0

Fixed all the tests in the yaml file

1a68442

added extra test case

29d16a2

merged main

71c3439

mridula-s109 requested a review from kderusso April 30, 2025 19:57

elasticsearchmachine and others added 2 commits April 30, 2025 20:04

[CI] Auto commit changes from spotless

819e279

Merge branch 'main' into pinned-retriever

bd29ce8

ioanatia approved these changes May 1, 2025

View reviewed changes

ioanatia reviewed May 1, 2025

View reviewed changes

kderusso approved these changes May 1, 2025

View reviewed changes

mridula-s109 merged commit b742b02 into main May 1, 2025
18 checks passed

mridula-s109 deleted the pinned-retriever branch May 1, 2025 12:50

elasticsearchmachine added the backport pending label May 1, 2025

mridula-s109 mentioned this pull request May 2, 2025

[8.19] Pinned retriever (#126401) #127639

Merged

nik9000 mentioned this pull request May 2, 2025

ESQL: Fix transport versions #127668

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Pinned retriever #126401

Pinned retriever #126401

mridula-s109 commented Apr 7, 2025 •

edited

Loading

elasticsearchmachine commented Apr 7, 2025

elasticsearchmachine commented Apr 7, 2025

elasticsearchmachine commented Apr 7, 2025

kderusso left a comment

elasticsearchmachine commented Apr 8, 2025

kderusso left a comment

mridula-s109 commented Apr 30, 2025

ioanatia left a comment

ioanatia May 1, 2025

kderusso May 1, 2025

kderusso left a comment

kderusso May 1, 2025

elasticsearchmachine commented May 1, 2025

mridula-s109 commented May 2, 2025

Pinned retriever #126401

Pinned retriever #126401

Conversation

mridula-s109 commented Apr 7, 2025 • edited Loading

Pinned Retriever for Elasticsearch

Key Features

Example API Request

Sample Response

elasticsearchmachine commented Apr 7, 2025

elasticsearchmachine commented Apr 7, 2025

elasticsearchmachine commented Apr 7, 2025

kderusso left a comment

Choose a reason for hiding this comment

elasticsearchmachine commented Apr 8, 2025

kderusso left a comment

Choose a reason for hiding this comment

mridula-s109 commented Apr 30, 2025

ioanatia left a comment

Choose a reason for hiding this comment

ioanatia May 1, 2025

Choose a reason for hiding this comment

kderusso May 1, 2025

Choose a reason for hiding this comment

kderusso left a comment

Choose a reason for hiding this comment

kderusso May 1, 2025

Choose a reason for hiding this comment

elasticsearchmachine commented May 1, 2025

💔 Backport failed

mridula-s109 commented May 2, 2025

💚 All backports created successfully

Questions ?

mridula-s109 commented Apr 7, 2025 •

edited

Loading