Fast and versatile tokenizer for language models, compatible with SentencePiece, Tokenizers, Tiktoken and more. Supports BPE, Unigram and WordPiece tokenization in JavaScript, Python and Rust.

Collection of GitHub Actions. 🧰

Mod manager for The Witcher 3 🗃

Benchmark for tokenizers. Utility to compare the performance of different tokenizers with different datasets.

Explore and compare the unstable features between versions of the Rust programming language

ESLint configuration presets in the flat config format

Packer for creating self-contained single-binary applications from executables and directories. Distribute your application without the need for an installer, with smaller file size and faster sta…

Resource Editor for parsing and modification of Windows Portable Executables, supporting icon replacement, resource transfer between files, and more.

SentencePiece model parser generated from the SentencePiece protobuf definition.

Procedural macro to embed files during compilation with optional compression

Utility to run commands with cross-platform environment variable expansion

Collection of hash implementations with const and no_std support

React package for Remix Icon 3 in a single tree-shakable module.

Minimal interface for the LZ4 compression library frame format

Simple tool to quickly count the frequency of words in a file or collection of files

New mod manager for The Witcher 3 🗃 (Incomplete)

General tokenizer library for the Web and Node. Supports Huggingface and Tiktoken formats