Open
Description
Porting BERTTokenizers enables several text embedding generation models. Requires #6988.
https://github.com/huggingface/text-embeddings-inference?tab=readme-ov-file#text-embeddings.
https://github.com/huggingface/transformers/blob/v4.37.0/src/transformers/models/bert/tokenization_bert.py#L137
We already have some BERT implementation which may be sufficient.