Language model

A statistical language model assigns a probability to a sequence of "m" words $P(w\_1,ldots,w\_m)$ by means of a probability distribution.

Language modeling is used in many natural language processing applications such as speech recognition, machine translation, part-of-speech tagging, parsing and information retrieval.

In speech recognition and in data compression, such a model tries to capture the properties of a language, and to predict the next word in a speech sequence.

When used in information retrieval, a language model is associated with a document in a collection. With query "Q" as input, retrieved documents are ranked based on the probability that the document's language model would generate the terms of the query, "P(Q|M_d)".

Estimating the probability of sequences can become difficult in corpora, in which phrases or sentences can be arbitrarily long and hence some sequences are not observed during training of the language model (data sparseness problem of overfitting). For that reason these models are often approximated using smoothed N-gram models.

N-gram models

In an n-gram model, the probability $P(w\_1,ldots,w\_m)$ of observing the sentence w₁,...,w_m is approximated as

$P(w\_1,ldots,w\_m)\; =\; prod^m\_\{i=1\}\; P(w\_i|w\_1,ldots,w\_\{i-1\})\; approx\; prod^m\_\{i=1\}\; P(w\_i|w\_\{i-(n-1)\},ldots,w\_\{i-1\})$

Here, it is assumed that the probability of observing the "i^th" word "w_i" in the context history of the preceding "i-1" words can be approximated by the probability of observing it in the shortened context history of the preceding "n-1" words ("n^th order Markov property).

The conditional probability can be calculated from n-gram frequency counts:$P(w\_i|w\_\{i-(n-1)\},ldots,w\_\{i-1\})\; =\; frac\{count(w\_\{i-(n-1)\},w\_\{i-1\},ldots,w\_i)\}\{count(w\_\{i-(n-1)\},ldots,w\_\{i-1\})\}$

The words bigram and trigram language model denote n-gram language models with "n=2" and "n=3", respectively.

Example

In a bigram (n=2) language model, the probability of the sentence "I saw the red house" is approximated as $P(I,saw,the,red,house)\; approx\; P(I)\; P(saw|I)\; P(the|saw)\; P(red|the)\; P(house|red)$

whereas in a trigram (n=3) language model, the approximation is$P(I,saw,the,red,house)\; approx\; P(I)\; P(saw|I)\; P(the|I,saw)\; P(red|saw,the)\; P(house|the,red)$

See also

* Factored language model

References

*cite conference | author=J M Ponte and W B Croft | url=http://citeseer.ist.psu.edu/ponte98language.html | title=A Language Modeling Approach to Information Retrieval | booktitle=Research and Development in Information Retrieval | year=1998 | pages=275-281
*cite conference | author=F Song and W B Croft | url=http://citeseer.ist.psu.edu/song99general.html | title=A General Language Model for Information Retrieval | booktitle=Research and Development in Information Retrieval | year=1999 | pages=279-280