Allow timeout during trained model download process

[dan-rubinstein]

Description

We currently allow users to provide a timeout during inference endpoint creation and when performing an inference request. When creating an endpoint requiring a trained model deployment to be started or performing an inference request to a default endpoint that does not have a trained model deployment started we will download the model before starting a deployment if it has not been previously downloaded. During this download process, we do not currently timeout if the user's requested timeout is exceeded and instead download the model fully and then timeout during the model deployment starting process. This change fixes this poor experience and allows the system to timeout during the model download. If this timeout occurs, we should still retain the experience that the model will be downloaded and a trained model deployment will be started in the background so the user does not have to take any further action for the process to complete.

Testing

• Tested that locally creating an ElasticsearchInternalService endpoint with a small timeout (1 second) will throw the ModelDeploymentTimeoutException and will complete the download/deployment start asynchronously.
• Tested that calling inference on a default endpoint with no model downloaded/no trained model deployment started has the same experience as the test above.
• Should we have some QA tests or IT tests for this?

TODO: Test what happens when an inference endpoint is created with a short timeout. It still downloads the model, creates the endpoint, and starts the deployment deployment but the error message is confusing as it tells the user to try again.

[elasticsearchmachine]

Hi @dan-rubinstein, I've created a changelog YAML for you.

[dan-rubinstein]

@elasticmachine merge upstream

[elasticsearchmachine]

Pinging @elastic/ml-core (Team:ML)