huggingface/tokenizers

A high-performance word segmentation library launched by Hugging Face, specially optimized for research and production environments. It supports a variety of models and hardware, and provides bindings for languages ​​such as Python for text preprocessing in natural language processing.