Improving Neural Topic Models using Knowledge Distillation

Understand

Topic models are often used to identify human-interpretable topics to help make sense of large document collections.

• We use knowledge distillation to combine the best attributes of probabilistic topic models and pretrained transformers.
• Our modular method can be straightforwardly applied with any neural topic model to improve topic quality, which we demonstrate using two models having disparate architectures, obtaining state-of-the-art topic coherence.
• We show that our adaptable framework not only improves performance in the aggregate over all estimated topics, as is commonly reported, but also in head-to-head comparisons of aligned topics.