2018

The WiLI benchmark dataset for written language identification

Thoma, Martin

Understand

This paper describes the WiLI-2018 benchmark dataset for monolingual written natural language identification.

WiLI-2018 is a publicly available, free of charge dataset of short text extracts from Wikipedia.
It contains 1000 paragraphs of 235 languages, totaling in 23500 paragraphs.
WiLI is a classification dataset: Given an unknown paragraph written in one dominant language, it has to be decided which language it is.

Built on

W. B. Cavnar and J. M. Trenkle, “N-gram-based text categorization,” Ann Arbor MI , vol. 48113, no. 2, pp. 161–175, 1994. [Online]. Available: http://odur.let.rug.nl/~vannoord/TextCat/textcat.pdf
1994
Earlier work this paper cites.
E. Jones, T. Oliphant, P. Peterson et al. , “SciPy: Open source scientific tools for Python,” 2001–. [Online]. Available: http://www.scipy.org/
2001
Earlier work this paper cites.
A. Pekker, “Source code for nltk.classify.textcat,” 2001. [Online]. Available: http://www.nltk.org/_modules/nltk/classify/textcat.html
2001
Earlier work this paper cites.
P. Koehn, “Europarl: A parallel corpus for statistical machine translation,” in MT summit , vol. 5, 2005, pp. 79–86
2005
Earlier work this paper cites.
D. Tran and D. Sharma, “Markov models for written language identification,” in Proceedings of the 12th international conference on neural information processing , 2005, pp. 67–70
2005
Earlier work this paper cites.
B. Hughes, T. Baldwin, S. Bird, J. Nicholson, and A. MacKinlay, “Reconsidering language identification for written language resources,” in Proc. International Conference on Language Resources and Evaluation , 2006, pp. 485–488
2006
Earlier work this paper cites.
B. Steven, E. Klein, and E. Loper, “Natural language processing with Python,” OReilly Media Inc , 2009. [Online]. Available: http://www.nltk.org
2009
Earlier work this paper cites.
T. Baldwin and M. Lui, “Language identification: The long and the short of the matter,” in Human Language Technologies: The 2010 Annual Conference of the North American Chapter of the Association for Computational Linguistics . Association for Computational Linguistics, 2010, pp. 229–237
2010
Earlier work this paper cites.
S. Nakatani, “Language detection library for Java,” slideshare.net, Dec. 2010. [Online]. Available: https://www.slideshare.net/shuyo/language-detection-library-for-java
2010
Earlier work this paper cites.
N. VRL, “Multilingual language identification: Altw 2010 shared task dataset,” in Australasian Language Technology Association Workshop 2010 , 2010, p. 4
2010
Earlier work this paper cites.

Similar

M. Lui and T. Baldwin, “Cross-domain feature selection for language identification,” in In Proceedings of 5th International Joint Conference on Natural Language Processing . Citeseer, 2011
2011
Cited alongside, same era.
F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, J. Vanderplas, A. Passos, D. Cournapeau, M. Brucher, M. Perrot, and E. Duchesnay, “Scikit-learn: Machine learning in Python,” Journal of Machine Learning Research , vol. 12, pp. 2825–2830, 2011
2011
Cited alongside, same era.
——, “langid. py: An off-the-shelf language identification tool,” in Proceedings of the ACL 2012 system demonstrations . Association for Computational Linguistics, 2012, pp. 25–30
2012
Cited alongside, same era.
H. F. Gaines, Cryptanalysis: a study of ciphers and their solution . Courier Corporation, 2014
2014
Cited alongside, same era.
J. Goldsmith, “wikipedia 1.4.0,” PyPI - the Python Package Index, Nov. 2014. [Online]. Available: https://pypi.python.org/pypi/wikipedia/
2014
Cited alongside, same era.
D. Kingma and J. Ba, “Adam: A method for stochastic optimization,” arXiv preprint arXiv:1412.6980 , Dec. 2014. [Online]. Available: https://arxiv.org/abs/1412.6980
Original
2014
Cited alongside, same era.
M. Lui, J. H. Lau, and T. Baldwin, “Automatic detection and language identification of multilingual documents,” Transactions of the Association for Computational Linguistics , vol. 2, pp. 27–40, 2014
2014
Cited alongside, same era.
T. Solorio, E. Blair, S. Maharjan, S. Bethard, M. Diab, M. Ghoneim, A. Hawwari, F. AlGhamdi, J. Hirschberg, A. Chang et al. , “Overview for the first shared task on language identification in code-switched data,” in Proceedings of the First Workshop on Computational Approaches to Code Switching , 2014, pp. 62–72. [Online]. Available: http://emnlp2014.org/workshops/CodeSwitch/call.html
2014
Cited alongside, same era.
M. Zampieri, L. Tan, N. Ljubešić, and J. Tiedemann, “Dsl shared task (finished),” 2014. [Online]. Available: http://corporavm.uni-koeln.de/vardial/sharedtask.html
2014
Cited alongside, same era.
M. Abadi, A. Agarwal, P. Barham, E. Brevdo, Z. Chen, C. Citro, G. S. Corrado, A. Davis, J. Dean, M. Devin, S. Ghemawat, I. Goodfellow, A. Harp, G. Irving, M. Isard, Y. Jia, R. Jozefowicz, L. Kaiser, M. Kudlur, J. Levenberg, D. Mané, R. Monga, S. Moore, D. Murray, C. Olah, M. Schuster, J. Shlens, B. Steiner, I. Sutskever, K. Talwar, P. Tucker, V. Vanhoucke, V. Vasudevan, F. Viégas, O. Vinyals, P. Warden, M. Wattenberg, M. Wicke, Y. Yu, and X. Zheng, “TensorFlow: Large-scale machine learning on heterogeneous systems,” 2015, software available from tensorflow.org. [Online]. Available: http://tensorflow.org/
2015
Cited alongside, same era.

Then

F. Chollet et al. , “Keras,” https://github.com/fchollet/keras , 2015
2015
Later among the works it cites.
——, “language-detection,” Dec. 2015. [Online]. Available: https://github.com/shuyo/language-detection/blob/wiki/ProjectHome.md
2015
Later among the works it cites.
A. Zubiaga, I. San Vicente, P. Gamallo, J. R. Pichel, I. Alegria, N. Aranberri, A. Ezeiza, and V. Fresno, “Tweetlid: a benchmark for tweet language identification,” Language Resources and Evaluation , vol. 50, no. 4, pp. 729–766, 2016
2016
Later among the works it cites.
“ISO 639-3 downloads,” Feb. 2017. [Online]. Available: http://www-01.sil.org/iso639-3/download.asp
2017
Later among the works it cites.
M. Lui, “langid.py,” Jul. 2017. [Online]. Available: https://github.com/saffsd/langid.py
2017
Later among the works it cites.
K. W. Mark Davis, “Unicode normalization forms,” 2017. [Online]. Available: http://unicode.org/reports/tr15/
2017
Later among the works it cites.
M. Thoma, “Analysis and optimization of convolutional neural network architectures,” Jul. 2017. [Online]. Available: https://arxiv.org/abs/1707.09725
Original
2017
Later among the works it cites.
——, “What is the name of (1 - sum of component-wise min)? Is it a metric?” Mathematics Stack Exchange, Aug. 2017. [Online]. Available: https://math.stackexchange.com/q/2409553
2017
Later among the works it cites.
——, “MartinThoma/clana: Initial alpha release of clana,” Jan. 2018. [Online]. Available: https://github.com/MartinThoma/clana
2018
Closest in time.
——, “MartinThoma/lidtk: Initial release,” Jan. 2018. [Online]. Available: https://github.com/MartinThoma/lidtk
2018
Closest in time.

Beyond the bibliography

alphaXiv searches the wider corpus for related work and actual follow-ups.

Open on alphaXiv

alphaXiv is searching for related work…

Reading the bibliography…