. ├── app.py ├── forms.py ├── models.py ├── templates/ │ ├── base.html │ ├── chat.html │ ├── login.html ...
A lightweight Rust library for training GPT-style BPE tokenizers. The tiktoken library is excellent for inference but doesn't support training. The HuggingFace tokenizers library supports training but ...