[Feature]: Better base64 to torch tenser

[noooop]

🚀 The feature, motivation and pitch

Support float32, float16, bfloat16, fp8_e4m3, fp8_e5m2 embed dtype in #26414

The following line of code will raise annoying UserWarning

torch.frombuffer(
    base64.b64decode(data["embedding"]), dtype=torch_dtype
).to(torch.float32)

UserWarning:

examples/online_serving/pooling/embedding_embed_dtype_client.py:49: UserWarning: The given buffer is not writable, and PyTorch does not support non-writable tensors. This means you can write to the underlying (supposedly non-writable) buffer using the tensor. You may want to copy the buffer to protect its data or make it writable before converting it to a tensor. This type of warning will be suppressed for the rest of this program. (Triggered internally at /pytorch/torch/csrc/utils/tensor_new.cpp:1578.)
  torch.frombuffer(

Cannot use numpy to load data['embedding'] because numpy does not support bfloat16, fp8_e4m3, fp8_e5m2

Welcome to contribute. If you know an efficient method that does not trigger this UserWarning (It's best to use a zero-copy method here.)

This issue involves the following files

• examples/online_serving/pooling/embedding_embed_dtype_client.py
• tests/entrypoints/pooling/openai/test_pooling.py
• tests/entrypoints/pooling/openai/test_embedding.py

Alternatives

No response

Additional context

No response

Before submitting a new issue...

• Make sure you already searched for relevant issues, and asked the chatbot living at the bottom right corner of the documentation page, which can answer lots of frequently asked questions.

[DarkLight1337]

cc @zou3519 any idea how to solve this? (this code is only used in examples/tests)

[noooop]

cc @zou3519 any idea how to solve this? (this code is only used in examples/tests)

It's used in examples, so I don't want users to see the annoying warning.

[ayushsatyam146]

@noooop @DarkLight1337 I can work on fixing this. Can you please assign it to me if that's okay?

[DarkLight1337]

Looks like someone beat you to it already

[DarkLight1337]

I'm not sure whether the fix is appropriate though. @zou3519 might know better

[noooop]

Looks like someone beat you to it already

This problem is not as simple as it seems LOL. It might require a new PyTorch API.

torch.frombuffer, although it has an annoying UserWarning, (I'm not 100% sure), I think it is a zero-copy method

[DarkLight1337]

From my understanding, this just means an error will occur if you try to mutate the tensor. But in the example this doesn't happen so it should be fine. Perhaps we should just suppress the UserWarning?

[noooop]

From my understanding, this just means an error will occur if you try to mutate the tensor. But in the example this doesn't happen so it should be fine. Perhaps we should just suppress the UserWarning?

It's not really a big issue; leave it here and see if the community has a better solution.

In fact, b64decode could also be a performance bottleneck because the clock speeds of the server CPUs are all very low.