Modelling Word Burstiness in Natural Language: A Generalised Polya Process for Document Language Models in Information Retrieval

Understand

We introduce a generalised multivariate Polya process for document language modelling.

• The framework outlined here generalises a number of statistical language models used in information retrieval for modelling document generation.
• In particular, we show that the choice of replacement matrix M ultimately defines the type of random process and therefore defines a particular type of document language model.
• We show that a particular variant of the general model is useful for modelling term-specific burstiness.