Lexicon

What Is a Lexicon?

A lexicon is the inventory of words and fixed expressions belonging to a language, a dialect, a technical field, or an individual speaker, together with the information a language user or a computer program needs in order to use each item correctly. It is distinguished from a grammar, which specifies how items combine, and from a dictionary, which is one particular published presentation of lexical information. In linguistics the lexicon is treated as a structured component of the language system rather than an unordered list, and in language technology it is a data resource with a defined schema, a coverage profile, and a maintenance cost.

The term is used at several scales. A language has a lexicon numbering in the hundreds of thousands of entries, a specialized domain such as cardiology or power systems has a controlled technical lexicon, and a speech recognition system has a pronunciation lexicon that may be deliberately restricted to a few thousand items for a given task.

Structure of a Lexical Entry

A lexical entry is keyed on a lemma, the citation form of a word, and carries several layers of annotation. Morphological information records inflectional class and derivational relations, so that "run," "runs," "ran," and "running" resolve to one entry. Syntactic information records the part of speech and the argument structure a word licenses, which is what tells a parser that "give" takes both a direct and an indirect object while "sleep" takes neither. Semantic information records the sense inventory, since most frequent words are polysemous, plus selectional preferences and relations to other senses. Phonological information supplies pronunciation, stress, and syllabification. Multiword expressions such as idioms and light verb constructions are stored as entries in their own right because their meaning cannot be composed from their parts.

Machine-Readable Lexical Resources

Computational work depends on lexicons that machines can traverse. WordNet, built at Princeton University beginning in the mid-1980s, organizes English nouns, verbs, adjectives, and adverbs into synsets, sets of synonymous senses, and links those synsets by hypernymy, meronymy, antonymy, and other relations. Its design was described in a 1995 Communications of the ACM article by George Miller, and it became the reference resource for word sense disambiguation and semantic similarity for two decades. Related resources encode different facets: FrameNet records semantic frames and the roles they evoke, PropBank annotates predicate argument structure over a treebank, and thesaurus-derived resources take a different organizing principle, as work on Roget's Thesaurus as a lexical resource for natural language processing demonstrates by comparing its category structure with WordNet's relation graph.

Coverage, Acquisition, and Statistical Alternatives

No fixed lexicon covers a living language. Word frequencies follow a Zipfian distribution, so a large fraction of the vocabulary in any new text consists of rare items, and proper names, neologisms, and technical terms arrive continuously. Traditional systems handled this with out-of-vocabulary fallbacks and morphological guessers. Contemporary systems more often sidestep the closed lexicon entirely, using subword tokenization such as byte pair encoding to represent any string from a fixed set of fragments, and using distributional vector representations learned from corpora in place of hand-authored sense inventories. Curated lexicons retain their value where interpretability and auditability matter: sentiment lexicons, terminology databases for regulated industries, and pronunciation dictionaries for speech synthesis are all still built and maintained by hand.

Applications

Lexicons have applications in a range of fields, including: