Skip to content

[ML] Adding placeholder functionality for custom model request logic #127271

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged

Conversation

jonathan-buttner
Copy link
Contributor

This PR adds logic to serialize java objects into JSON values and implement the placeholder replacement logic. The Custom service was leveraging gson to handle the serialization into json. This implementation uses our internal XContent provider instead.

This is needed for the Custom service functionality in the inference API. The Custom service will support PUT requests with placeholders (${some_key}). The ValidatingSubstitutor handles replacing the placeholders with the values passed in via a map.

Example PUT Request

PUT _inference/sparse_embedding/custom_sparse_embedding
{
  "service":"custom-model",
  "service_settings":{
    "secret_parameters":{
      "api_key":<<your api_key>>
    },
    "url":"http://url",
    "headers":{
      "Authorization": "Bearer ${api_key}",
      "Content-Type": "application/json;charset=utf-8"
    },
    "request":{
      "content":"""
        {
          "input": ${input},
          "input_type": "${input_type}",
          "return_token": ${return_token}
        }
        """
    },
    "response":{
      "json_parser":{
        "token_path":"$.result[*].embeddings[*].token",
         "weight_path":"$.result[*].embeddings[*].weight"
      }
    }
  },
  "task_settings":{
    "parameters":{
      "input_type":"query",
      "return_token":true
    }
  }
}

In this example ${input_type} would be replaced with query and ${return_token} would be replaced with true.

@jonathan-buttner jonathan-buttner added >non-issue :ml Machine learning Team:ML Meta label for the ML team auto-backport Automatically create backport pull requests when merged v8.19.0 v9.1.0 labels Apr 23, 2025
@@ -57,9 +57,11 @@ dependencies {
implementation 'io.grpc:grpc-context:1.49.2'
implementation 'io.opencensus:opencensus-api:0.31.1'
implementation 'io.opencensus:opencensus-contrib-http-util:0.31.1'
implementation "org.apache.commons:commons-lang3:${versions.commons_lang3}"
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Both of these are needed for the Apache commons StringSubstitutor being used for replacing the placeholder.

public static <T> String toJson(T value, String field) {
try {
XContentBuilder builder = JsonXContent.contentBuilder();
builder.value(value);
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This handles the quoting for us, for example a string will be surrounded by quotes.

@jonathan-buttner jonathan-buttner marked this pull request as ready for review April 23, 2025 18:58
@elasticsearchmachine
Copy link
Collaborator

Pinging @elastic/ml-core (Team:ML)

}

private static void ensureNoMorePlaceholdersExist(String substitutedString, String field) {
Matcher matcher = VARIABLE_PLACEHOLDER_PATTERN.matcher(substitutedString);
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Wouldn't this just be substitutedString.indexOf('$') != -1? Or do we allow instances of $ (like at the end of a string)?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yeah we'd have to look for ${ I suppose, but yeah I think the idea would be to let a string have any number of $ throughout it or even ${ (as long as it didn't have the closing }).

@jonathan-buttner jonathan-buttner merged commit 6bb79f2 into elastic:main Apr 24, 2025
16 of 17 checks passed
@jonathan-buttner jonathan-buttner deleted the ml-custom-model-placeholder branch April 24, 2025 14:03
jonathan-buttner added a commit to jonathan-buttner/elasticsearch that referenced this pull request Apr 24, 2025
…lastic#127271)

* Adding placeholder functionality for request logic

* Updating comments

* Fixing tests

* Adding missing licenses

* Fixing String.format
@elasticsearchmachine
Copy link
Collaborator

💚 Backport successful

Status Branch Result
8.x

elasticsearchmachine pushed a commit that referenced this pull request Apr 24, 2025
…127271) (#127329)

* Adding placeholder functionality for request logic

* Updating comments

* Fixing tests

* Adding missing licenses

* Fixing String.format
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
auto-backport Automatically create backport pull requests when merged :ml Machine learning >non-issue Team:ML Meta label for the ML team v8.19.0 v9.1.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants