Tokenizer — AI Glossary

A tokenizer is the component that converts raw text into a sequence of tokens (numerical IDs) that an AI model can process, and converts model output tokens back into readable text. Different models use different tokenization schemes — for example, Byte Pair Encoding (BPE) or SentencePiece — which affects how text is split, how many tokens a given text consumes, and how the model handles different languages.