What Is Tokenization?

Tokenization is the process of dividing a sequence of input data into discrete units called tokens, which are then processed or stored in place of the original data. The term spans two related but distinct technical domains. In natural language processing (NLP), tokenization converts raw text into a sequence of words, subwords, or characters that a computational system can analyze. In data security, tokenization replaces sensitive data values, such as payment card numbers, with non-sensitive substitute tokens that retain no exploitable information. Both uses share the underlying idea of converting a continuous or sensitive input into a structured set of representational units, though the goals and mechanisms differ substantially.

Tokenization entered NLP as a foundational preprocessing operation alongside the development of information retrieval and corpus linguistics in the 1960s and 1970s. The security sense of the term emerged as payment processing systems sought alternatives to storing raw card data in merchant environments. Both domains have grown substantially in technical complexity as the underlying applications have scaled.

Text Tokenization in Natural Language Processing

In NLP, tokenization is defined as the task of segmenting a character sequence into tokens, typically words or punctuation units, while discarding whitespace and applying normalization rules. The Stanford NLP textbook distinguishes a token (a specific instance in a document) from a type (the abstract class of identical tokens) and a term (a normalized type that appears in the system's dictionary). Tokenization decisions are load-bearing for downstream tasks: a query and the documents it retrieves must be tokenized by the same rules, or matches will fail. Language-specific challenges include handling possessives and contractions in English, compound words in German, and character-based segmentation in Mandarin, where word boundaries are not marked in the writing system.

Subword Tokenization Methods

Modern large language models use subword tokenization algorithms rather than simple word splitting, because word-level vocabularies grow too large to be practical and fail on out-of-vocabulary words. Byte Pair Encoding (BPE), WordPiece, and Unigram Language Model tokenization all segment text into fragments smaller than full words, building a fixed-size vocabulary that covers the vast majority of text encountered in training and inference. GPT models use a variant of BPE; BERT and its descendants use WordPiece. These methods represent a practical resolution to the vocabulary-size tradeoff: a 50,000-token BPE vocabulary covers most English text with rare words decomposed into recognizable subword units. The design of tokenization schemes for multilingual and code-mixed text remains an active research area, with work appearing regularly in IEEE Transactions on Neural Networks and Learning Systems.

Data Security Tokenization

In security applications, tokenization replaces a sensitive data element, most commonly a payment card Primary Account Number (PAN), with a randomly generated surrogate value called a token. The original value is stored in a separate, controlled token vault; the token itself carries no mathematical relationship to the original and cannot be reversed without access to the vault. The PCI Security Standards Council's tokenization guidelines describe how properly implemented tokenization removes card numbers from merchant environments, substantially reducing the scope of systems that must comply with Payment Card Industry Data Security Standard (PCI DSS) requirements. Unlike encryption, tokenized data cannot be decrypted even by an attacker who obtains the token, because no key exists for the mapping outside the vault.

Applications

Tokenization has applications in a wide range of fields, including:

• Text preprocessing for search engines, machine translation, and large language models
• Payment processing security and PCI DSS compliance for merchants and processors
• Healthcare data de-identification for sharing clinical records without patient identifiers
• Database security for protecting personally identifiable information in enterprise systems
• Compiler design, where source code is tokenized as the first stage of the compilation pipeline