Skip to content

Expose EOS and BOS ids in tokenizer APIs #955

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
ankan-ban opened this issue May 18, 2025 · 0 comments
Open

Expose EOS and BOS ids in tokenizer APIs #955

ankan-ban opened this issue May 18, 2025 · 0 comments

Comments

@ankan-ban
Copy link

In most LLM inference applications the token generation loop needs to know the EOS token id to figure out when to stop. The tokenizer already has this information, but it doesn't expose any API that the caller can use to retrieve it.

This results in applications/callers either hardcoding the EOS./BOS ids or having to resort to other metadata (e.g, the ONNXRuntime GenAI sdk has to duplicate this information in GenAI config file).

It would be great if such an API can be exposed from ORTX Tokenizer itself.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant