Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model#7425

Merged

tarekgh merged 3 commits intodotnet:mainfrom

tarekgh:SupportByteLevelEncodingInBpeTokenizer

Mar 19, 2025

Commits on Mar 18, 2025

Support ByteLevel encoding in Bpe tokenizer to support DeepSeek model
tarekgh
committed
Update src/Microsoft.ML.Tokenizers/Model/BpeOptions.cs

tarekgh
and
Copilot
authored

Commits on Mar 19, 2025

Feedback 1
tarekgh
committed