Resolve unsupported InferenceRequest parameters in BLS

[tonywang10101]

Description

This PR addresses the issue where the VLLM tutorial cannot run in BLS mode. The problem was initially pointed out in the pb_stub.cc file, line 1426.

Changes

Moved parameters to the input field instead of using the parameter field in InferenceRequest.

[the-david-oy]

Stellar contribution! Thanks for submitting this, Tony. Can you submit the Contributor License Agreement (instructions here) so that we can review this?

CC: @tanmayv25 @rmccorm4

[the-david-oy]

Tested locally. This passes the vLLM example.

[tanmayv25]

@nnshah1 Do you think moving the sampling parameters from request parameters to input should be fine? Looks like it should be useful for BLS as well as PA profiling? Note that perf_analyzer currently do not support request parameters.
cc @matthewkotila

[debermudez]

It is in active development so the decision here will impact our final PR.

[tonywang10101]

Stellar contribution! Thanks for submitting this, Tony. Can you submit the Contributor License Agreement (instructions here) so that we can review this?

CC: @tanmayv25 @rmccorm4

Thanks @dyastremsky. I have signed the CLA and mailed it.

[the-david-oy]

Thank you for getting that done so quickly, Tony! We had a discussion and would like to support both the parameter and tensor paths. Since this PR removes support for the parameter path, would you be able to amend it to support both?

Quoting @rmccorm4's suggestion:

So, we can add the tensor, but should include optional: true in the model config for that tensor.
In the model, if the tensor is provided, use it. If not provided, use the parameters.
If both are provided, either error or join the 2 dicts? Not sure. Joining is probably a better experience with a doc note somewhere mentioning it.
In the client, we can demonstrate both passing as parameters or tensor. Maybe we add a --mode {parameters, tensor} with default="tensor" to control it?

[nnshah1]

@tonywang10101 to ease merging with support for parameters and tensors I created this PR on your original. Can you take a look and merge?

tonywang10101#1

[the-david-oy]

Hi Tony! Thanks for all your work here.

This PR has been merged with you and Neelay as co-authors here: #52