Unified tokenizer type onboarding#1540

Open

srikary12 wants to merge 7 commits intopytorch:mainfrom

srikary12:Tokenizer-New-Type-Onboarding

Commits on May 11, 2025

Unified tokenizer type onboarding
srikary12
committed

Commits on May 13, 2025

Merge branch 'main' into Tokenizer-New-Type-Onboarding
zhenyan-zhang-meta
authored

Commits on May 14, 2025

Refactor tokenizer type handling to use Enum directly and remove redundant methods
srikary12
committed
Merge branch 'Tokenizer-New-Type-Onboarding' of https://github.com/srikary12/torchchat into Tokenizer-New-Type-Onboarding
srikary12
committed
Merge branch 'main' into Tokenizer-New-Type-Onboarding
zhenyan-zhang-meta
authored

Commits on May 25, 2025

Merge branch 'main' into Tokenizer-New-Type-Onboarding
srikary12
authored

Commits on May 27, 2025

Modify export to use unified tokenizer type
srikary12
committed