speechbrain.tokenizersΒΆ

Package defining the SentencePiece tokenizer

speechbrain.tokenizers.SentencePiece

Library for Byte-pair-encoding (BPE) tokenization. Authors * Abdelwahab Heba 2020 * Loren Lugosch 2020.