2021

One Size Does Not Fit All: Finding the Optimal Subword Sizes for FastText Models across Languages

Novotný, Vít, Ayetiran, Eniafe Festus, Bačovský, Dalibor et al.

Understand

Unsupervised representation learning of words from large multilingual corpora is useful for downstream tasks such as word sense disambiguation, semantic text similarity, and information retrieval.

  • The representation precision of log-bilinear fastText models is mostly due to their use of subword information.
  • In previous work, the optimization of fastText's subword sizes has not been fully explored, and non-English fastText models were trained using subword sizes optimized for English and German word analogy tasks.
  • In our work, we find the optimal subword sizes on the English, German, Czech, Italian, Spanish, French, Hindi, Turkish, and Russian word analogy tasks.

Reading the bibliography…